Audio assessment for analyzing sleep trends using machine learning techniques

ABSTRACT

A computer system for assessing sound to analyze a user&#39;s sleep includes a processor configured to perform operations including: (i) storing sample sound data associated with a plurality of sample sleep events, the sample sound data including a plurality of sample characteristics each respectively associated with at least one sample sleep event of the plurality of sample sleep events; (ii) receiving, from a client device, subject sound data collected during a sleep interval; (iii) analyzing, using a machine learning algorithm, the subject sound data collected during the sleep interval; (iv) identifying, based upon the analyzing, a subject characteristic associated with the subject sound data; (v) comparing the subject characteristic with the plurality of sample characteristics; and (vi) determining, based upon the comparing, whether the subject characteristic substantially matches at least one sample characteristic to identify one or more subject sleep events occurring during the sleep interval.

FIELD OF THE INVENTION

The present disclosure relates to systems and methods for audioassessment to analyze sleep trends. More particularly, the presentdisclosure relates to systems and methods for analyzing a person's sleeppatterns using machine learning techniques.

BACKGROUND

Many people experience issues during their sleep, whether due to adiagnosed or diagnosable condition such as sleep apnea or insomnia, ordue to unknown conditions or environmental factors, such as sleeping inan area that experiences a lot of noise. However, it can often bedifficult to identify a source or cause of poor sleep without undergoingtime-consuming, uncomfortable, and/or expensive tests. These tests areoften undertaken in a lab rather than in a person's home, which caneasily result in the person being unable to sleep comfortably or notexhibiting the same personal or environmental issues with their sleep.In such cases, issues can go unresolved or undiagnosed.

BRIEF SUMMARY

The present embodiments relate to systems and methods for assessingsound data collected during a sleep interval to identify one or moresleep events that occur during that sleep interval. More particularly,the present embodiments relate to using machine learning to comparesound data from the sleep interval to one or more sample sounds. Bymatching characteristics of the sound data collected during the sleepinterval to characteristics of the sample sounds, one or more sleepevents occurring during the interval may be identified.

In one aspect, a computer-based method for assessing sound to analyze auser's sleep is provided. The method may be performed using a soundanalysis computing device including at least one processor incommunication with at least one memory device. The method may include:(i) storing sample sound data associated with a plurality of samplesleep events, the sample sound data including a plurality of samplecharacteristics each respectively associated with at least one samplesleep event of the plurality of sample sleep events; (ii) receiving,from a client device, subject sound data collected during a sleepinterval; (iii) analyzing, using a machine learning algorithm, thesubject sound data collected during the sleep interval; (iv)identifying, based upon the analyzing, a subject characteristicassociated with the subject sound data; (v) comparing the subjectcharacteristic with the plurality of sample characteristics; and/or (vi)determining, based upon the comparing, whether the subjectcharacteristic substantially matches at least one sample characteristicto identify one or more subject sleep events occurring during the sleepinterval. The method may include additional, fewer, and/or alternativesteps, including those described elsewhere herein.

In another aspect, a computer system for assessing sound to analyze auser's sleep is described. The computer system includes a processor, anda non-transitory, tangible, computer-readable storage medium havinginstructions stored thereon that, in response to execution by theprocessor, cause the processor to perform operations including: (i)storing sample sound data associated with a plurality of sample sleepevents, the sample sound data including a plurality of samplecharacteristics each respectively associated with at least one samplesleep event of the plurality of sample sleep events; (ii) receiving,from a client device, subject sound data collected during a sleepinterval; (iii) analyzing, using a machine learning algorithm, thesubject sound data collected during the sleep interval; (iv)identifying, based upon the analyzing, a subject characteristicassociated with the subject sound data; (v) comparing the subjectcharacteristic with the plurality of sample characteristics; and/or (vi)determining, based upon the comparing, whether the subjectcharacteristic substantially matches at least one sample characteristicto identify one or more subject sleep events occurring during the sleepinterval. The computer system may include additional, less, and/oralternative functionality, including that described elsewhere herein.

In yet another aspect, a non-transitory computer readable medium thatincludes executable instructions for assessing sound to analyze a user'ssleep is described. When executed by a sound analysis computing deviceincluding at least one processor in communication with at least onememory device, the computer executable instructions may cause the soundanalysis computing device to: (i) store sample sound data associatedwith a plurality of sample sleep events, the sample sound data includinga plurality of sample characteristics each respectively associated withat least one sample sleep event of the plurality of sample sleep events;(ii) receive, from a client device, subject sound data collected duringa sleep interval; (iii) analyze, using a machine learning algorithm, thesubject sound data collected during the sleep interval; (iv) identify,based upon the analysis, a subject characteristic associated with thesubject sound data; (v) compare the subject characteristic with theplurality of sample characteristics; and/or (vi) determine, based uponthe comparison, whether the subject characteristic substantially matchesat least one sample characteristic to identify one or more subject sleepevents occurring during the sleep interval. The computer readable mediummay include additional, fewer, and/or alternative instructions,including instructions to perform processes described elsewhere herein.

Advantages will become more apparent to those skilled in the art fromthe following description of the preferred embodiments which have beenshown and described by way of illustration. As will be realized, thepresent embodiments may be capable of other and different embodiments,and their details are capable of modification in various respects.Accordingly, the drawings and description are to be regarded asillustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The Figures described below depict various aspects of the systems andmethods disclosed therein. It should be understood that each Figuredepicts an embodiment of a particular aspect of the disclosed systemsand methods, and that each of the Figures is intended to accord with apossible embodiment thereof. Further, wherever possible, the followingdescription refers to the reference numerals included in the followingFigures, in which features depicted in multiple Figures are designatedwith consistent reference numerals.

There are shown in the drawings arrangements which are presentlydiscussed, it being understood, however, that the present embodimentsare not limited to the precise arrangements and are instrumentalitiesshown, wherein:

FIG. 1 illustrates a schematic diagram of an exemplary computer systemfor analyzing sound data associated with a user's sleep.

FIG. 2 illustrates an exemplary configuration of a client device shownin FIG. 1 , in accordance with one embodiment of the present disclosure.

FIG. 3 illustrates an exemplary configuration of a server shown in FIG.1 , in accordance with one embodiment of the present disclosure.

FIG. 4 is a data flow diagram illustrating the flow of data betweencomponents of the computer system shown in FIG. 1 .

FIG. 5 illustrates an exemplary process implemented by the computersystem shown in FIG. 1 for assessing sound to analyze a user's sleepusing machine learning techniques.

The Figures depict preferred embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the systems and methodsillustrated herein may be employed without departing from the principlesof the invention described herein.

DETAILED DESCRIPTION OF THE DRAWINGS

The present embodiments may relate to, inter alia, systems and methodsfor analyzing a person's sleep patterns using audio assessment andmachine learning techniques. In one exemplary embodiment, the processmay be performed by at least one front-end system, such as a clientdevice (e.g., a smart phone), and at least one back-end system, such asa web server and/or a database server.

Accordingly, the system may include a client device, such as a personalcomputer or a mobile communications device. The user may connect to theback-end system via the client device to initiate a process for audioassessment to analyze sleep patterns. More particularly, the backendsystem may deliver a webpage or a mobile application (e.g., an “app”) tothe client device. The webpage or app may measure sound data capturedduring a sleep interval, corresponding generally to an interval of timeover which a user is sleeping (or, if a level of sleep quality is low,attempting to get to sleep or remain asleep). In addition, and asdescribed below, the backend system and/or the smartphone app maycompare one or more characteristics of sound data collected during thesleep interval (“subject characteristics”) to one or morecharacteristics associated with known sleep events (“samplecharacteristics”) to identify one or more sleep events that occurredduring the sleep interval (“subject sleep events”). Such comparison mayprovide the user insight into one or more sleep conditions or issues theuser is experiencing, which may facilitate improving the user's sleep.In addition, the webpage or app may enable the user to enter varioususer inputs, such as user-identified sleep events, qualitative opinionson sleep quality, and/or other user inputs.

In various embodiments, a backend system (e.g., a web server and/or adatabase server) and/or a client device may analyze sound dataassociated with a plurality of known sleep events, also referred to as“sample sleep events.” Sleep events may include, for instance, sleepdisorders (e.g., sleep apnea, insomnia, sleep movement disorder,snoring, parasomnia, etc.), sleep stages (e.g., falling asleep, lightsleep, rapid eye movement (REM) sleep, waking), environmental factors orconditions (e.g., white noise, weather sounds, etc.), and/or otherevents that may occur during a user's sleep interval. This sample sounddata may be stored in a database and may include digital and/or analogsound data associated with the plurality of sample sleep events.

The analysis may be performed based upon one or more machine learningalgorithms (as described below), such as one or more machine learningalgorithms configured to identify one or more characteristics associatedwith each of the sample sleep events. The stored sample sound data mayinclude a plurality of sound files associated with respective samplesleep events. In one embodiment, a user associated with training themachine learning algorithm(s), referred to as an “expert user,” mayindicate the content of each sound file. For instance, the expert usermay indicate that one set of sound files contains sample sound data ofsleep condition sleep events, such as a person experiencing sleep apnea.

Each of these different sample sleep events may be associated with oneor more sound characteristics, such as frequency characteristics,amplitude characteristics, various other waveform characteristics,duration characteristics, and the like. The sample characteristics maybe identified by the system (e.g., by training and/or implementing oneor more machine learning algorithms) or may be identified by an expertuser to the system (e.g., during a training phase for the machinelearning algorithm). For example, the expert user may identify one ormore characteristics of a sleep apnea sleep event (i.e., the sound(s)made during a sleep apnea sleep event). Additionally or alternatively,the expert user may indicate the content of a sound file of the samplesleep event, and the system may “self-learn” one or more characteristicsassociated with that sample sleep event. Thus, the system may identify,based upon an analysis of various sound data associated with one or moresleep events, one or more characteristics, such as at least a first“sample characteristic,” associated with one or more sample sleepevents. Where a particular sample sleep event is associated with aplurality of sample characteristics, these sample characteristics maycollectively be referred to as a unique sample sound “signature” orsample sleep event signature. It should be readily understood thatcertain sample characteristics may be common to a plurality of thesample sleep events (e.g., particular breathing patterns or levels), butthat sample sound signatures include a particular set of characteristicsthat uniquely identify that sample sleep event.

During or after the sleep interval, the system may, in addition,receive, from the client device (which may be disposed or located withina threshold distance from the user during the sleep interval), sounddata, such as ambient sound data, collected by the client device withina user's sleeping area (e.g., within a user's bedroom). Moreparticularly, a smartphone app, as described above, running on theclient device may “listen” to or detect the sound within the sleepingarea. The client device may, in addition (and as described above)include an analog to digital converter, which may convert analog sounddata collected by the client device to digital sound data. This sounddata collected during the sleep interval may be referred to herein as“subject sound data,” to be compared to sample sound data associatedwith the sample sleep events, as described above.

In response to receiving the subject sound data from the client device,the system may analyze the subject sound data. More particularly, thesystem may analyze the subject sound data collected during the sleepinterval to identify one or more characteristics, such as at least afirst “subject characteristic,” of the sound data collected during thesleep interval. These subject characteristics may include frequencycharacteristics, amplitude characteristics, various other waveformcharacteristics, and the like, associated with the user's sleep duringthe sleep interval. Further, where a plurality of subjectcharacteristics are identified in conjunction with the subject sounddata, the sound characteristics may collectively define a subject soundsignature.

In addition, in some cases, the system may identify a plurality ofgroups of characteristics or a plurality of sound signatures within thesubject sound data, such that subject sound data collected during thesleep interval actually includes a plurality of sound signatures. Thismay occur, for example, as a result of a plurality of distinct sleepevents occurring during the sleep interval, such as a first sleep eventassociated with disrupted sleep and a second sleep event associated witha thunderstorm. Certain sounds may, in other words, intermingle orintermix with one or more other sounds during the sleep interval. Thesystem may, under such a circumstance, identify characteristicsassociated with both sounds, such that the thunderstorm during the sleepinterval is associated with a first subject characteristic, a firstgroup of subject characteristics, or a first subject sound signature,and the sound of the disrupted sleep is associated with a second subjectcharacteristic, a second group of subject characteristics, or a secondsubject sound signature.

In response to identifying at least one subject characteristicassociated with the subject sound data collected during the sleepinterval, the system may compare the at least one subject characteristicto at least one sample characteristic of the plurality of sample sleepevents (or a subset thereof). For example, the system may compare one ormore frequency characteristics and/or one or more amplitudecharacteristics of the subject sound data collected during the sleepinterval to one or more frequency characteristics and/or one or moreamplitude characteristics of the sample sound data associated with oneor more sample sleep events. If the comparison yields a match, thesystem may determine that the subject sound data includes frequency,amplitude, or other waveform characteristics of a particular samplesleep event, which may suggest or indicate that the user experienced anoccurrence of that particular sleep event during the sleep interval.Where the user experiences an occurrence of a sleep event, thatexperienced sleep event is referred to as a “subject sleep event,” whichis generally identified by matching the subject characteristic(s) of thesubject sleep event to the sample characteristic(s) of the sample sleepevent. Thus, the system may determine, based upon such a comparison andthe resulting match, that the user experienced a subject sleep eventduring the sleep interval.

In some embodiments, the system may determine that the user experienceda subject sleep event (i.e., may identify the occurrence of the subjectsleep event) based upon a partial or substantial match between at leastone subject characteristic of the subject sound data collected duringthe sleep interval and at least one sample characteristic of the samplesound data associated with a sample sleep event. As used herein, a“substantial” match may be made between at least one subjectcharacteristic of subject sound data collected during the sleep intervaland at least one characteristic of sample sound data associated with asample sleep event when the characteristics are similar (e.g., within apredefined range of each other) but not exactly the same.

The system may define a particular threshold range associated with asubstantial match. If the characteristics fall within the predefinedthreshold range of one another, then the system may identify asubstantial match. The threshold range may be variable based uponcertain factors, such as user inputs. For instance, if a user of theclient device indicates that a storm occurred during the sleep interval(e.g., using the app running on the client device), a threshold rangeassociated with a storm sample sleep event (and the samplecharacteristics thereof) may be adjusted (e.g., expanded). In such acircumstance, the system may be more likely to identify a storm sleepevent within the subject sound data. Additionally or alternatively, thesystem may access other data to define the threshold ranges forparticular sleep events. For instance, the system may access weatherdata to determine the weather during a particular sleep interval and mayadjust the threshold range based on the weather data. In other words,the system may leverage user-input data or otherwise accessed data toadjust the threshold range for any particular sleep event, or type ofsleep event.

Further, in some embodiments, the system may compare a plurality offrequency characteristics and/or a plurality of amplitudecharacteristics of the sound data collected during the sleep interval(i.e., subject characteristics) to a plurality of frequencycharacteristics and/or a plurality of amplitude characteristics of thesound data associated with sample sleep events (i.e., samplecharacteristics). Specifically, the system may compare a subject soundsignature associated with the sound data collected during the sleepinterval to a sample sound signature of a sample sleep event. Inaddition, in some cases, the system may compare a subject soundsignature associated with the sound data collected during the sleepinterval to a plurality of sample sound signatures associated with arespective plurality of sample sleep events. If the subject soundsignature associated with the sound data collected during the sleepinterval matches, or substantially matches, at least one sample soundsignature of a sample sleep event, the system may determine that theuser experienced that sleep event during the sleep interval.

After one or more subject sleep events are identified as occurringduring the sleep interval, the system may perform additional analysesassociated with the sleep interval. The system may determine a durationof each identified subject sleep event. For instance, the system maydetermine an initial time at which the subject sleep event wasoccurring, based upon the initiation of the subject sound signatureassociated with that subject sleep event. The system may furtherdetermine a terminal time at which the subject sleep event stopped,based upon the termination of the subject sound signature associatedwith that subject sleep event. The duration of each subject sleep eventmay be no longer than the sleep interval.

The system may additionally or alternatively determine a number ofoccurrences of a subject sleep event. For instance, the system maydetermine that a subject sleep event recurs during the sleep interval(e.g., is identified more than once according to the methods describedabove), such as the user being restless, waking, snoring, etc. Thesystem may then determine how many times that particular subject sleepevent occurred. Additionally or alternatively, the system may determinea number of occurrences of a type or class of subject sleep event, suchas how many times sleep events associated with waking or being restlessoccurred, how many times sleep events associated with sleep-breathingconditions occurred, and/or how many times environmental sleep events(e.g., thunderclaps, HVAC systems activating, etc.) occurred.

In some embodiments, the system may perform one or more additionalanalyses to identify trends or correlations associated with subjectsleep events experienced during one or more sleep intervals. Forinstance, the system may determine that a plurality of “restless”-typesleep events were generally correlated with (e.g., occurredsimultaneously or within a threshold amount of time of one another) aplurality of environmental-type sleep events, such as thunderclaps orother environmental noise. As another example, the system may processsound data (e.g., sound data that has already been analyzed or “raw”sound data) for a plurality of generally subsequent sleep intervals,such as a plurality of sleep intervals occurring over a week or a month.The system may analyze the occurrence(s) of various subject sleep eventsto identify one or more trends in the user's sleep. The system maydetermine, for instance, (i) that fewer sleep events occur onweeknights, (ii) that the user is experiencing a greater number of aparticular sleep event, or (iii) that the user experiences a higher rateof sleep events when the sleep interval is shorter (e.g., less than sixhours). Moreover, the system may process user input in conjunction withthe sound data to identify additional trends or correlations. Forexample, the user may input to the system (e.g., using the smartphoneapp) that they exercise on certain days, and the system may determinethat fewer restless-type sleep events are experienced on those days.

The system may determine a level of sleep quality based upon theidentified sleep events. The system may use one or more algorithms,equations, models, etc., to quantify the user's level of sleep qualityusing the identified sleep events as inputs. For example, a highernumber of identified sleep events occurring during a single sleepinterval may result in a lower calculated or determined level of sleepquality. The system may account for different types or classes of sleepevents in determining the level of sleep quality. For instance,restless-type sleep events may contribute to a lower level of sleepquality, whereas deep-sleep-type events (e.g., regular breathing, lowactivity or movement) may contribute to a higher level of sleep quality,and environmental-type sleeps events may not be factored in or may beweighted lower in determining the level of sleep quality. The system mayuse other factors in determining the level of sleep quality, including,for example, but not limited to, a duration of the sleep interval and/orof individual sleep events, start or end time(s) of the sleep interval,user-input data identifying subjective opinions of sleep quality, etc.

The system may be configured to leverage user input to refine themachine learning algorithms. For example, a user may use the clientdevice (e.g., the app running on the client device or the webpageaccessed thereby) to indicate that one or more sleep events occurredduring a sleep interval. The user may wake up in the middle of the sleepinterval and provide user input indicating that they are awake. In sucha circumstance, the system may associate one or more subjectcharacteristics at the time of the waking with a “waking” sleep event.The user may identify a weather event that occurred during the night,such as a storm or excessive wind gusts. In such a circumstance, thesystem may adjust a threshold range for matching such weather events,such that the system is more likely to interpret subject characteristicsfrom the subject sound data in accordance with the user's input (e.g.,as associated with weather events). Moreover, the system may use anyidentified subject sleep event, in particular those confirmed by theuser to have occurred, as a sample sleep event to refine the machinelearning algorithm. Where the system has identified a subject sleepevent—a sleep event experienced by the user—the system may then storesubject sound data associated with the subject sleep event as samplesound data, which the system may then use to identify future occurrencesof that sleep event. In one particular example, the system may use theprocesses described above and identify a sleep apnea sleep event duringa sleep interval. As described herein, the system may generate anotification or recommendation that the user see a sleep specialist toconfirm that the user is, in fact, experiencing sleep apnea. If thesleep specialist confirms that the subject sleep event was a sleep apneasleep event, the system may store the subject sound data associated withthat identified sleep apnea sleep event as a sample sleep apnea sleepevent. Accordingly, when the system compares future subject sound datawith the stored sample sound data, the system may more preciselyidentify sleep apnea sleep events by having a greater number of sleepapnea sample sounds to compare against.

In some embodiments, the system may develop a sleep pattern profile forthe user that includes such details as typical or common breathingpatterns for the user during different phases of sleep (e.g., fallingasleep, deep sleep, waking, etc.). The system may not storeuser-specific data as sample sleep data that the system uses to“self-train” the machine learning algorithms, where the machine learningalgorithms are applied to more than one user. If the system implements auser-specific version of the machine learning algorithm, the system maystore user-specific data, including sleep pattern profiles, and leveragethe user-specific data to refine the user-specific machine learningalgorithm.

In certain embodiments, the system may be configured to analyze wakingbehavior alongside sound data collected during a sleep interval. Forinstance, the client device (e.g., using the smartphone app) may beconfigured to collect motion data during the user's waking hours thatmay then be analyzed for correlation with sleep events. In oneparticular embodiment, driving behavior data may be collected using theclient device and analyzed in conjunction with the sound data collectedduring a sleep interval. The system may determine one or more trends orcorrelations between the driving behavior data and the sound data (orone or more sleep events identified therein). For instance, the systemmay determine that a lower quality of sleep of a sleep interval ishighly correlated with more risky driving behavior data the followingday (e.g., sharper cornering, more hard-braking, etc.). As anotherexample, the client device (e.g., using the smartphone app) may beconfigured to collect sound and/or motion data, such as breathingpattern data, during the user's waking hours. The system may leverage auser's sleep pattern profile to analyze the user's waking behaviors. Forexample, the sleep pattern profile may include breathing patterns orother sounds associated with the user falling asleep. Based uponbreathing pattern data collected during the user's waking hours, ascompared to the sleep pattern profile, the system may determine that theuser is falling asleep. If the system determines that the user isperforming certain tasks, such as driving (e.g., using motion data), thesystem may transmit a control instruction to the client device todeliver a visual and/or audible alarm to prevent the user from fallingasleep, for instance, while driving.

The system may, in addition, generate and provide a notification to theclient device and/or to another device associated with the user oranother party (e.g., a healthcare provider) that one or more subjectsleep events occurring during the sleep interval were identified. Forexample, the system may provide the notification to the client device ata time after an end of the sleep interval (such that the notificationdoes not disrupt the sleep interval). The notification may be formattedin any suitable format for receipt at the client device (or anothercomputing device), such as an in-app message, an email message, a textmessage, a voice message, and/or any other visual and/or audiblenotification. The notification may include a description of each subjectsleep event identified as occurring during the sleep interval. Thenotification may further describe a duration and/or number ofoccurrences of each identified subject sleep event. Additionally oralternatively, the notification may provide an indicator of sleepquality experienced by the user during the sleep interval, as describedabove. The notification may provide other information about the sleepinterval, such as a date of the sleep interval, a beginning and end timeof the sleep interval, weather information associated with the sleepinterval, other environmental information (e.g., a temperature withinthe sleeping area, whether or not a white noise machine was activated,whether or not the user was using a sleep aid, whether or not the userwas using a CPAP machine, etc.), and/or any other information determinedby the system or provided by the user. The notification may additionallyor alternatively include advice or recommendations based upon theanalyses performed by the system. For instance, if the system determineda correlation between restless-type sleep events and environmental-typesleep events (as described above), the system may generate arecommendation that the user take steps to create a sound barrier toreduce the effects of the environmental-type sleep events, such as usingear plugs, using a white noise machine, or sleeping in a moresound-proofed sleeping area.

In some embodiments, the notification may be provided in associationwith an insurance policy or other insurance-associated notification andmay include recommendations or advice that associates the user's sleepwith the user's driving behavior. In such embodiments, the notificationmay provide recommendations that the user more carefully assess theirdriving behavior on days following a sleep interval with a shortduration or low level of sleep quality.

In some embodiments, the notification may be provided in associationwith a healthcare plan or other healthcare-associated notification andmay include recommendations or information about how sleep can affectthe user's health. The notification may include, for instance, arecommendation to visit a sleep specialist based upon the identificationof one or more subject sleep events associated with sleep apnea oranother sleep condition.

Exemplary technical effects of the systems, methods, andcomputer-readable media described herein may include, for example: (a)training of a machine learning algorithm, such as a neural network, toidentify sleep events and characteristics thereof for audio analysis ofsleep sound data; and/or (b) reducing or eliminating the need for in-labsleep analysis.

Exemplary System for Audio Assessment and Sound Identification toAnalyze Sleep Sound Data

FIG. 1 depicts a view of an exemplary system 100 for audio assessmentand sound identification to analyze sound data collected during a sleepinterval. In one exemplary embodiment, system 100 may include a clientdevice, such as a client device 102. Client device 102 may be associatedwith an individual, such as a user who is experiencing trouble sleeping.System 100 may also include network 104, a web server 106, a databaseserver 108, and/or a database 110.

Accordingly, in the exemplary, client device 102 may be any personalcomputing device and/or any mobile communications device of a user, suchas a personal computer, a tablet computer, a smartphone, and the like.Client device 102 may, as described below, include one or more audiosensors (e.g., microphones) and may be configured to present anapplication (e.g., a smartphone “app”), which may be configured toreceive, or “listen to” sound, such as ambient noise, within a sleepingarea (not shown in FIG. 1 ) during a sleep interval. To this end, clientdevice 102 may include or execute software for viewing and interactingwith a smartphone app that receives or listens to sound within thesleeping area.

Network 104 may be any electronic communications system, such as anycomputer network or collection of computer networks, and may incorporatevarious hardware and/or software. Communication over network 104 may beaccomplished via any suitable communication channels, such as, forexample, one or more telephone networks, one or more extranets, one ormore intranets, the Internet, one or more point of interaction devices(e.g., point of sale devices, smart phones, cellular phones), variousonline and/or offline communications systems, such as various local areaand wide area networks, and the like.

Web server 106, which may also be referred to as a sound analysiscomputing device 106, may be any computer or computer system that isconfigured to receive and process data, such as sound data, transmittedby and received from client device 102. Web server 106 may be coupledbetween client device 102 and database server 108. More particularly,web server 106 may be communicatively coupled to client device 102 vianetwork 104. In various embodiments, web server 106 may be directlycoupled to database server 108 and/or communicatively coupled todatabase server 108 via a network, such as network 104. Web server 106may, in addition, function to store, process, and/or deliver one or moreweb pages and/or any other suitable content to client device 102. Webserver 106 may, in addition, receive data, such as sound data providedto the smartphone app (as described herein) and/or a webpage from clientdevice 102 for subsequent transmission to database server 108.

In various embodiments, web server 106 may implement various hardwareand/or software, such as, for example, one or more communicationprotocols, one or more message brokers, one or more data processingengines, one or more servlets, one or more application servers, and thelike. For instance, in one embodiment, web server 106 may implement anInternet of Things (IoT) protocol, such as a machine-to-machine IoTcommunications protocol (e.g. an MQTT protocol). In addition, in variousembodiments, web server 106 may implement a message broker programmodule configured to translate a message or communications from amessaging protocol of a sending device to a messaging protocol of areceiving device (e.g., RABBITTMQ, KAFKA, ACTIVEMQ, KESTREL). Furtherstill, in some embodiments, web server 106 may implement a dataprocessing engine, such as a cluster computing framework like APACHESPARK. In addition, in various embodiments, web server 106 may implementservlet and/or JSP server, such as APACHE TOMCAT.

Database server 108 may be any computer or computer program thatprovides database services to one or more other computers or computerprograms. In various embodiments, database server 108 may becommunicatively coupled between web server 108 and database 110.Database server 108 may, in addition, function to process data receivedfrom web server 106, such as sound data, which may include, for example,sound data received from client device 102.

In various embodiments, web server 106 and/or database server 108 maytrain and/or implement one or more machine learning algorithms, asdescribed herein, to process sound data, such as sound data storedwithin database 110. For example, in some embodiments, web server 106and/or database server 108 may process sound data associated with aknown or “sample” sleep events, to identify one or more samplecharacteristics associated with each sample sleep event. Such samplecharacteristics may include any characteristics or features associatedwith the respective sample sleep event, such as frequencycharacteristics, amplitude characteristics, and/or any other waveformcharacteristics of the sound data (which may be digital or analog) ofone or more sample sleep events. In some embodiments, web server 106and/or database server 108 may identify a sample sound “signature” or asample sound “fingerprint” associated with one or more sample sleepevents. A sample sound signature of sample sleep event may include aplurality of waveform or other characteristics associated with thesample sleep event and may function to uniquely identify the samplesleep event.

Database 110 may be any organized collection of data, such as, forexample, any data organized as part of a relational data structure, anydata organized as part of a flat file, and the like. Database 110 may becommunicatively coupled to database server 108 and may receive datafrom, and provide data to, database server 108, such as in response toone or more requests for data, which may be provided via a databasemanagement system (DBMS) implemented on database server 108. In variousembodiments, database 110 may be a non-relational database, such as anAPACHE HADOOP database.

Although the components of system 100 are described below and depictedat FIG. 1 as being interconnected in a particular configuration, it iscontemplated that the systems, subsystems, hardware and softwarecomponents, various network components, and database systems describedherein may be variously configured and interconnected and maycommunicate with one another within system 100 to facilitate theprocesses and advantages described herein. For example, although asingle web server 106, a single database server 108, and a singledatabase 110 are described above, it will be appreciated that system 100may include any suitable number of interconnected, communicativelycoupled, web servers, database servers, and/or databases. Further,although certain functions, processes, and operations are describedherein with respect to one or more system components, it is contemplatedthat one or more other system components may perform the functions,processes, and operations described herein.

Exemplary Client Device

FIG. 2 depicts an exemplary configuration of a client device 202, suchas client device 102, as shown in FIG. 1 , and in accordance with oneembodiment of the present disclosure. Client device 202 may be operatedby a user 201. Client device 202 may include a processor 205 forexecuting instructions. In some embodiments, executable instructions maybe stored in a memory area 210. Processor 205 may include one or moreprocessing units (e.g., in a multi-core configuration). Memory area 210may be any device allowing information such as executable instructionsand/or sound data to be stored and retrieved. Memory area 210 mayinclude one or more computer readable media.

Client device 202 may also include at least one media output component215 for presenting information to user 201. Media output component 215may be any component capable of conveying information to user 201. Insome embodiments, media output component 215 may include an outputadapter (not shown) such as a video adapter and/or an audio adapter. Anoutput adapter may be operatively coupled to processor 205 and adaptedto operatively couple to an output device such as a display device(e.g., a cathode ray tube (CRT), liquid crystal display (LCD), lightemitting diode (LED) display, or “electronic ink” display) or an audiooutput device (e.g., a speaker or headphones).

In some embodiments, media output component 215 may be configured topresent a graphical user interface (e.g., a web browser and/or a clientapplication) to user 201. A graphical user interface may include, forexample, an online store interface for viewing and/or purchasing items,and/or a wallet application for managing payment information. In someembodiments, client device 202 may include an input device 220 forreceiving input from user 201. User 201 may use input device 220 to,without limitation, select and/or enter data, such as, for example,subjective opinions of sleep quality, “diary” entries of behavior orevents during waking hours, etc.

Input device 220 may include, for example, a keyboard, a pointingdevice, a mouse, a stylus, a touch sensitive panel (e.g., a touch pad ora touch screen), a gyroscope, an accelerometer, a position detector, abiometric input device, and/or an audio input device or audio sensor. Asingle component such as a touch screen may function as both an outputdevice of media output component 215 and input device 220.

Client device 202 may also include a communication interface 225,communicatively coupled via network 104 to web server 106 (shown in FIG.1 ). Communication interface 225 may include, for example, a wired orwireless network adapter and/or a wireless data transceiver for use witha mobile telecommunications network.

Stored in memory area 210 are, for example, computer readableinstructions for providing a user interface to user 201 via media outputcomponent 215 and, optionally, receiving and processing input from inputdevice 220. A user interface may include, among other possibilities, aweb browser and/or a client application. Web browsers enable users, suchas user 201, to display and interact with media and other informationtypically embedded on a web page or a website.

Client device 202 may also include one or more audio sensors, such asaudio sensor 230. Audio sensor 230 may be any suitable audio sensor,such as a microphone, for receiving and/or collecting sound data duringa sleep interval. Audio sensor 230 may be communicatively coupled toprocessor 205, which may implement an analog to digital converter insoftware to convert analog sound data received by audio sensor 230 todigital data. In some embodiments, client device 202 may include aseparate (e.g., hardware and/or software) analog to digital convertercoupled between audio sensor 230 and processor 205. Moreover, in someembodiments, audio sensor 230 may be separate from and communicativelycoupled to client device 202. For instance, audio sensor 230 may beembodied as a separate microphone in wired connection with or wirelesslyconnected to client device 202.

Exemplary Database System

FIG. 3 depicts an exemplary database system 300 such as database server108 and database 110, as shown in FIG. 1 , and in accordance with oneexemplary embodiment of the present disclosure. Accordingly, databasesystem 300 may include a server computer device 301 (e.g., web server106 or database server 108, shown in FIG. 1 ), which may, in turn,include a processor 305 for executing instructions. Instructions may bestored in a memory area 310. Processor 305 may include one or moreprocessing units (e.g., in a multi-core configuration).

Processor 305 may be operatively coupled to a communication interface315 such that server computer device 301 is capable of communicatingwith a remote computing device, as described above. For example,communication interface 315 may receive requests from client device 202via the Internet and/or over a computer network.

Processor 305 may also be operatively coupled to a storage device 325(e.g., database 110). Storage device 325 may be any computer-operatedhardware suitable for storing and/or retrieving data, such as, but notlimited to, data associated with database 110. In some embodiments,storage device 325 may be integrated in server computer device 301. Forexample, server computer device 301 may include one or more hard diskdrives as storage device 325.

In other embodiments, storage device 325 may be external to servercomputer device 301 and may be accessed by a plurality of servercomputer devices 301. For example, storage device 325 may include astorage area network (SAN), a network attached storage (NAS) system,and/or multiple storage units such as hard disks and/or solid statedisks in a redundant array of inexpensive disks (RAID) configuration.

In some embodiments, processor 305 may be operatively coupled to storagedevice 325 via a storage interface 320. Storage interface 320 may be anycomponent capable of providing processor 305 with access to storagedevice 325. Storage interface 320 may include, for example, an AdvancedTechnology Attachment (ATA) adapter, a Serial ATA (SATA) adapter, aSmall Computer System Interface (SCSI) adapter, a RAID controller, a SANadapter, a network adapter, and/or any component providing processor 305with access to storage device 325.

Exemplary Process for Monitoring Sound to Analyze a User's Sleep UsingMachine Learning Techniques

FIG. 4 is a data flow diagram 400 illustrating the flow of data betweencomponents of the computer system 100 illustrated in FIG. 1 ,specifically to and from a sound analysis computing device 402, whichmay include and/or be similar to web server 106 (shown in FIG. 1 ). Inthe exemplary embodiment, computer system 100 includes sound analysiscomputing device 402. Sound analysis computing device 402 may includeand/or be similar to web server 106, such that sound analysis computingdevice 402 represents a back-end computing device of computer system100. Alternatively, sound analysis computing device 402 may includeclient device 102. In other words, the sound analysis processesdescribed herein may be implemented on client device 102, eithercompletely and/or by accessing web server 106 and/or another back-endcomputing device (e.g., to access one or more machine learningalgorithms). In the illustrated embodiment, sound analysis computingdevice 402 is separate from and in communication with client device 102(e.g., over network 104, shown in FIG. 1 ).

Specifically, client device 102 is located within a sleeping area 404(e.g., a bedroom or other sleeping area) of a user 406 (which may beuser 201, shown in FIG. 2 ) during one or more sleep intervals of user406. In particular, client device 102 is located within a thresholddistance of user 406 during the sleep interval, such that client device102 may capture sound data 408 during the sleep interval. Client device102 may be running an app (and/or may be providing a webpage) thatfacilitates collection of sound data 408 during a sleep interval andsubsequent transmittal of sound data 408 to sound analysis computingdevice 402. The app may facilitate additional functionality, such aspermitting user 406 to enter user input (as described herein) andenabling user 406 to receive and view notifications.

Sound analysis computing device 402 may compare one or morecharacteristics of sound data 408 collected during the sleep interval(“subject characteristics”) to one or more characteristics associatedwith known sleep events (“sample characteristics”) to identify one ormore sleep events that occur during the sleep interval (“subject sleepevents”). Such comparison may provide user 406 insight into one or moresleep conditions or issues user 406 is experiencing, which mayfacilitate improving the user's sleep.

In the exemplary embodiment, sound analysis computing device 402 mayanalyze sample sound data 410 associated with a plurality of known sleepevents, also referred to as “sample sleep events.” Sample sound data 410may be stored in a memory 412 of sound analysis computing device 402.Memory 412 may include and/or be similar to database 110 (shown in FIG.1 ). Sample sleep events may include, for instance, sleep disorders(e.g., sleep apnea, insomnia, sleep movement disorder, snoring,parasomnia, etc.), sleep stages (e.g., falling asleep, light sleep,rapid eye movement (REM) sleep, waking), environmental factors orconditions (e.g., white noise, weather sounds, etc.), and/or otherevents that may occur during a user's sleep interval. Sample sound data410 may include digital and/or analog sound data associated with theplurality of sample sleep events.

Sound analysis computing device 402 may perform the processes describedherein using one or more machine learning algorithms 414, such as one ormore machine learning algorithms 414 configured to identify one or morecharacteristics associated with each of the sample sleep events. Storedsample sound data 410 may include a plurality of sound files associatedwith respective sample sleep events. In one embodiment, a userassociated with training machine learning algorithm(s) 414, referred toas an “expert user,” may indicate the content of each sound file. Forinstance, the expert user may indicate that one set of sound filescontains sample sound data 410 of sleep condition sleep events, such asa person experiencing sleep apnea.

Each sample sleep event may be associated with one or morecharacteristics, such as frequency characteristics, amplitudecharacteristics, various other waveform characteristics, durationcharacteristics, and the like. The sample characteristics may beidentified by sound analysis computing device 402 (e.g., by trainingand/or implementing one or more machine learning algorithms 414) or maybe identified by the expert user to sound analysis computing device 402(e.g., during a training phase for machine learning algorithms 414). Forexample, the expert user may identify one or more characteristics of asleep apnea sleep event (i.e., the sound(s) made during a sleep apneasleep event). Additionally or alternatively, the expert user mayindicate the content of a sound file of the sample sleep event, andsound analysis computing device 402 may “self-learn” one or morecharacteristics associated with that sample sleep event. Thus, soundanalysis computing device 402 may identify, based upon an analysis ofsample sound data 410 associated with one or more sleep events, one ormore characteristics, such as at least a first “sample characteristic,”associated with one or more sample sleep events. Where a particularsample sleep event is associated with a plurality of samplecharacteristics, these sample characteristics may collectively bereferred to as a unique sample sound “signature” or sample sleep eventsignature. It should be readily understood that certain samplecharacteristics may be common to a plurality of the sample sleep events(e.g., particular breathing patterns or levels), but that sample soundsignatures include a particular set of characteristics that uniquelyidentify that sample sleep event.

During or after a sleep interval, sound analysis computing device 402may, in addition, receive, from client device 102, sound data 408 (alsoreferred to as “subject sound data”), such as ambient sound data,collected by client device 102 within sleeping area 404. Moreparticularly, a smartphone app, as described above, running on clientdevice 102 may “listen” to or detect the sound within sleeping area 404.Client device 102 may, in addition, include an analog to digitalconverter, which may convert analog sound data collected by clientdevice 102 to digital subject sound data 408.

In response to receiving subject sound data 408 from client device 102,sound analysis computing device 402 may analyze subject sound data 408.Sound analysis computing device 402 may include one or more analysiscomponent(s) 416 configured to perform these analyses. Analysiscomponent 416 may include specific computer-executable instructionscausing sound analysis computing device 402 to perform the processesdescribed herein. Alternatively, analysis component 416 may include aseparate processing component specifically programmed to perform theprocesses described herein. More particularly, sound analysis computingdevice 402 may analyze subject sound data 408 collected during the sleepinterval to identify one or more characteristics, such as at least afirst “subject characteristic,” of subject sound data 408. These subjectcharacteristics may include frequency characteristics, amplitudecharacteristics, various other waveform characteristics, and the like,associated with the user's sleep during the sleep interval. Further,where a plurality of subject characteristics are identified inconjunction with subject sound data 408, the sound characteristics maycollectively define a subject sound signature.

In addition, in some cases, sound analysis computing device 402 mayidentify a plurality of groups of characteristics or a plurality ofsound signatures within subject sound data 408, such that subject sounddata 408 actually includes a plurality of sound signatures. This mayoccur, for example, as a result of a plurality of distinct sleep eventsoccurring during the sleep interval, such as a first sleep eventassociated with disrupted sleep and a second sleep event associated witha thunderstorm. Certain sounds may, in other words, intermingle orintermix with one or more other sounds during the sleep interval. Soundanalysis computing device 402 may, under such a circumstance, identifycharacteristics associated with both sounds, such that the thunderstormduring the sleep interval is associated with a first subjectcharacteristic, a first group of subject characteristics, or a firstsubject sound signature, and the sound of the disrupted sleep isassociated with a second subject characteristic, a second group ofsubject characteristics, or a second subject sound signature.

In response to identifying at least one characteristic associated withsubject sound data 408, sound analysis computing device 402 may comparethe at least one subject characteristic to at least one samplecharacteristic of the plurality of sample sleep events (or a subsetthereof). For example, sound analysis computing device 402 may compareone or more frequency characteristics and/or one or more amplitudecharacteristics of subject sound data 408 to one or more frequencycharacteristics and/or one or more amplitude characteristics of samplesound data 410 associated with one or more sample sleep events. If thecomparison yields a match, sound analysis computing device 402 maydetermine that subject sound data 408 includes frequency, amplitude, orother waveform characteristics of a particular sample sleep event, whichmay suggest or indicate that user 406 experienced an occurrence of thatparticular sleep event during the sleep interval. Where user 406experiences an occurrence of a sleep event, that experienced sleep eventis referred to as a “subject sleep event,” which is generally identifiedby matching the subject characteristic(s) of the subject sleep event tothe sample characteristic(s) of the sample sleep event. Thus, soundanalysis computing device 402 may determine, based upon such acomparison and the resulting match, that user 406 experienced a subjectsleep event during the sleep interval.

In some embodiments, sound analysis computing device 402 may determinethat the user experienced a subject sleep event (i.e., may identify theoccurrence of the subject sleep event) based upon a partial orsubstantial match between at least one subject characteristic of subjectsound data 408 and at least one sample characteristic of sample sounddata 410. As used herein, a “substantial” match may be made between atleast one subject characteristic of subject sound data 408 and at leastone characteristic of sample sound data 410 when the characteristics aresimilar (e.g., within a predefined range of each other) but not exactlythe same.

Sound analysis computing device 402 may define a particular thresholdrange associated with a substantial match. If the characteristics fallwithin the predefined threshold range of one another, then soundanalysis computing device 402 may identify a substantial match. Thethreshold range may be variable based upon certain factors, such as userinputs. For instance, if user 406 of client device 102 indicates that astorm occurred during the sleep interval (e.g., using the app running onclient device 102), a threshold range associated with a storm samplesleep event (and the sample characteristics thereof) may be adjusted(e.g., expanded). In such a circumstance, sound analysis computingdevice 402 may be more likely to identify a storm sleep event withinsubject sound data 408. Additionally or alternatively, sound analysiscomputing device 402 may access other data to define the thresholdranges for particular sleep events. For instance, sound analysiscomputing device 402 may access weather data to determine the weatherduring a particular sleep interval and may adjust the threshold rangebased on the weather data. In other words, sound analysis computingdevice 402 may leverage user-input data or otherwise accessed data toadjust the threshold range for any particular sleep event, or type ofsleep event.

Further, in some embodiments, sound analysis computing device 402 maycompare a plurality of frequency characteristics and/or a plurality ofamplitude characteristics of subject sound data 408 (i.e., subjectcharacteristics) to a plurality of frequency characteristics and/or aplurality of amplitude characteristics of sample sound data 410 (i.e.,sample characteristics). Specifically, sound analysis computing device402 may compare a subject sound signature associated with subject sounddata 408 to a sample sound signature of a sample sleep event. Inaddition, in some cases, sound analysis computing device 402 may comparea subject sound signature associated with subject sound data 408 to aplurality of sample sound signatures associated with a respectiveplurality of sample sleep events. If the subject sound signatureassociated with subject sound data 408 matches, or substantiallymatches, at least one sample sound signature of a sample sleep event,sound analysis computing device 402 may determine that user 406experienced that sleep event during the sleep interval.

After one or more subject sleep events are identified as occurringduring the sleep interval, sound analysis computing device 402 mayperform additional analyses associated with the sleep interval. Soundanalysis computing device 402 may determine a duration of eachidentified subject sleep event. For instance, sound analysis computingdevice 402 may determine an initial time at which the subject sleepevent was occurring, based upon the initiation of the subject soundsignature associated with that subject sleep event. Sound analysiscomputing device 402 may further determine a terminal time at which thesubject sleep event stopped, based upon the termination of the subjectsound signature associated with that subject sleep event. The durationof each subject sleep event may be no longer than the sleep interval.

Sound analysis computing device 402 may additionally or alternativelydetermine a number of occurrences of a subject sleep event. Forinstance, sound analysis computing device 402 may determine that asubject sleep event recurs during the sleep interval (e.g., isidentified more than once according to the methods described above),such as the user being restless, waking, snoring, etc. Sound analysiscomputing device 402 may then determine how many times that particularsubject sleep event occurred. Additionally or alternatively, soundanalysis computing device 402 may determine a number of occurrences of atype or class of subject sleep event, such as how many times sleepevents associated with waking or being restless occurred, how many timessleep events associated with sleep-breathing conditions occurred, and/orhow many times environmental sleep events (e.g., thunderclaps, HVACsystems activating, etc.) occurred.

In some embodiments, sound analysis computing device 402 may perform oneor more additional analyses to identify trends or correlationsassociated with subject sleep events experienced during one or moresleep intervals. For instance, sound analysis computing device 402 maydetermine that a plurality of “restless”-type sleep events weregenerally correlated with (e.g., occurred simultaneously or within athreshold amount of time of one another) a plurality ofenvironmental-type sleep events, such as thunderclaps or otherenvironmental noise. As another example, sound analysis computing device402 may process subject sound data 408 collected during a plurality ofgenerally subsequent sleep intervals, such as a plurality of sleepintervals occurring over a week or a month. Sound analysis computingdevice 402 may analyze the occurrence(s) of various subject sleep eventsto identify one or more trends in the user's sleep. Sound analysiscomputing device 402 may determine that fewer sleep events occur onweeknights, that user 406 is experiencing a greater number of aparticular sleep event, or that user 406 experiences a higher rate ofsleep events when the sleep interval is shorter (e.g., less than sixhours). Moreover, sound analysis computing device 402 may process userinput in conjunction with subject sound data 408 to identify additionaltrends or correlations. For example, user 406 may input to soundanalysis computing device 402 (e.g., using the smartphone app) that theyexercise on certain days, and sound analysis computing device 402 maydetermine that fewer restless-type sleep events are experienced on thosedays.

Sound analysis computing device 402 may determine a level of sleepquality based upon the identified sleep events. Sound analysis computingdevice 402 may use one or more algorithms, equations, models, etc., toquantify the user's level of sleep quality using the identified sleepevents as inputs. For example, a higher number of identified sleepevents occurring during a single sleep interval may result in a lowercalculated or determined level of sleep quality. Sound analysiscomputing device 402 may account for different types or classes of sleepevents in determining the level of sleep quality. For instance,restless-type sleep events may contribute to a lower level of sleepquality, whereas deep-sleep-type events (e.g., regular breathing, lowactivity or movement) may contribute to a higher level of sleep quality,and environmental-type sleeps events may not be factored in or may beweighted lower in determining the level of sleep quality. Sound analysiscomputing device 402 may use other factors in determining the level ofsleep quality, including, for example, but not limited to, a duration ofthe sleep interval and/or of individual sleep events, start or endtime(s) of the sleep interval, user-input data identifying subjectiveopinions of sleep quality, etc.

Sound analysis computing device 402 may be configured to leverage userinput to refine machine learning algorithms 414. For example, user 406may use client device 102 (e.g., the app running on client device 102 orthe webpage accessed thereby) to indicate that one or more sleep eventsoccurred during a sleep interval. User 406 may wake up in the middle ofthe sleep interval and provide user input indicating that they areawake. In such a circumstance, sound analysis computing device 402 mayassociate one or more subject characteristics at the time of the wakingwith a “waking” sleep event. User 406 may identify a weather event thatoccurred during the night, such as a storm or excessive wind gusts. Insuch a circumstance, sound analysis computing device 402 may adjust athreshold range for matching such weather events, such that soundanalysis computing device 402 is more likely to interpret subjectcharacteristics from subject sound data 408 in accordance with theuser's input (e.g., as associated with weather events). Moreover, soundanalysis computing device 402 may use any identified subject sleepevent, in particular those confirmed by user 406 to have occurred, as asample sleep event to refine machine learning algorithms 414. Wheresound analysis computing device 402 has identified a subject sleepevent—a sleep event experienced by user 406—sound analysis computingdevice 402 may then store subject sound data 408 associated with thesubject sleep event as sample sound data 410, which sound analysiscomputing device 402 may then use to identify future occurrences of thatsleep event. In one particular example, sound analysis computing device402 may use the processes described above and identify a sleep apneasleep event during a sleep interval. As described herein, sound analysiscomputing device 402 may generate a notification or recommendation thatuser 406 see a sleep specialist to confirm that user 406 is, in fact,experiencing sleep apnea. If the sleep specialist confirms that thesubject sleep event was a sleep apnea sleep event, sound analysiscomputing device 402 may store subject sound data 408 associated withthat identified sleep apnea sleep event as a sample sleep apnea sleepevent. Accordingly, when sound analysis computing device 402 comparesfuture subject sound data (not specifically shown) with stored samplesound data 410, sound analysis computing device 402 may more preciselyidentify sleep apnea sleep events by having a greater number of sleepapnea sample sounds to compare against.

In some embodiments, sound analysis computing device 402 may develop asleep pattern profile for user 406 that includes such details as typicalor common breathing patterns for user 406 during different phases ofsleep (e.g., falling asleep, deep sleep, waking, etc.). Sound analysiscomputing device 402 may not store user-specific data as sample sleepdata 410 that sound analysis computing device 402 uses to “self-train”machine learning algorithms 414, where machine learning algorithms 414are applied to more than one user. If sound analysis computing device402 implements a user-specific version of machine learning algorithm414, sound analysis computing device 402 may store user-specific data(e.g., as sample sound data 410), including sleep pattern profiles, andleverage the user-specific data to refine the user-specific machinelearning algorithm 414.

In certain embodiments, sound analysis computing device 402 may beconfigured to analyze waking behavior alongside subject sound data 408.For instance, client device 102 (e.g., using the smartphone app) may beconfigured to collect motion data (not shown) during the user's wakinghours that may then be analyzed for correlation with sleep events. Inone particular embodiment, driving behavior data (not shown) may becollected using client device 102 and analyzed in conjunction withsubject sound data 408. Sound analysis computing device 402 maydetermine one or more trends or correlations between the drivingbehavior data and subject sound data 408 (or one or more sleep eventsidentified therein). For instance, sound analysis computing device 402may determine that a lower quality of sleep of a sleep interval ishighly correlated with more risky driving behavior data the followingday (e.g., sharper cornering, more hard-braking, etc.). As anotherexample, client device 102 (e.g., using the smartphone app) may beconfigured to collect sound and/or motion data, such as breathingpattern data, during the user's waking hours. Sound analysis computingdevice 402 may leverage a user's sleep pattern profile to analyze theuser's waking behaviors. For example, the sleep pattern profile mayinclude breathing patterns or other sounds associated with user 406falling asleep. Based upon breathing pattern data collected during theuser's waking hours, as compared to the sleep pattern profile, soundanalysis computing device 402 may determine that user 406 is fallingasleep. If sound analysis computing device 402 determines that user 406is performing certain tasks, such as driving (e.g., using motion data),sound analysis computing device 402 may transmit a control instructionto client device 102 to deliver a visual and/or audible alarm to preventuser 406 from falling asleep, for instance, while driving.

Sound analysis computing device 402 may, in addition, generate andprovide a notification 418 to client device 102 and/or (with userconsent) to a third-party device 420 associated with another party(e.g., a healthcare provider) that one or more subject sleep eventsoccurring during the sleep interval were identified. For example, soundanalysis computing device 402 may provide notification 418 to clientdevice 102 at a time after an end of the sleep interval (such that thenotification does not disrupt the sleep interval). Notification 418 maybe formatted in any suitable format for receipt at client device 102 (orthird-party device 420), such as an in-app message, an email message, atext message, a voice message, and/or any other visual and/or audiblenotification. Notification 418 may include a description of each subjectsleep event identified as occurring during the sleep interval.Notification 418 may further describe a duration and/or number ofoccurrences of each identified subject sleep event. Additionally oralternatively, notification 418 may provide an indicator of sleepquality experienced by user 406 during the sleep interval, as describedabove. Notification 418 may provide other information about the sleepinterval, such as a date of the sleep interval, a beginning and end timeof the sleep interval, weather information associated with the sleepinterval, other environmental information (e.g., a temperature withinsleeping area 404, whether or not a white noise machine was activated,whether or not user 406 was using a sleep aid, whether or not user 406was using a CPAP machine, etc.), and/or any other information determinedby sound analysis computing device 402 or provided by user 406.

Notification 418 may additionally or alternatively include advice orrecommendations based upon the analyses performed by sound analysiscomputing device 402. For instance, if sound analysis computing device402 determined a correlation between restless-type sleep events andenvironmental-type sleep events (as described above), sound analysiscomputing device 402 may generate a recommendation that user 406 takesteps to create a sound barrier to reduce the effects of theenvironmental-type sleep events, such as using ear plugs, using a whitenoise machine, or sleeping in a more sound-proofed sleeping area 404.

In some embodiments, notification 418 may be provided in associationwith an insurance policy or other insurance-associated notification andmay include recommendations or advice that associates the user's sleepwith the user's driving behavior. In such embodiments, notification 418may provide recommendations that user 406 more carefully assess theirdriving behavior on days following a sleep interval with a shortduration or low level of sleep quality.

In some embodiments, notification 418 may be provided in associationwith a healthcare plan or other healthcare-associated notification andmay include recommendations or information about how sleep can affectthe user's health. Notification 418 may include, for instance, arecommendation to visit a sleep specialist based upon the identificationof one or more subject sleep events associated with sleep apnea oranother sleep condition.

FIG. 5 depicts a flowchart of an exemplary process 500 for assessingsound to analyze a user's sleep using machine learning techniques,process 500 implemented by computer system 100, such as web server 106,and/or by sound analysis computing device 402.

In the exemplary embodiment, process 500 may include storing 502 samplesound data (e.g., sample sound data 410, shown in FIG. 4 ) associatedwith a plurality of sample sleep events, the sample sound data includinga plurality of sample characteristics each respectively associated withat least one sample sleep event of the plurality of sample sleep events.Process 500 may also include receiving 504, from a client device (e.g.,client device 102, shown in FIG. 1 ), subject sound data (e.g., subjectsound data 408, shown in FIG. 4 ) collected during a sleep interval.Process 500 may include analyzing 506 the subject sound data collectedduring the sleep interval, and identifying 508, based upon analyzing506, a subject characteristic associated with the subject sound data.Process 500 may still further include comparing 510 the subjectcharacteristic with the plurality of sample characteristics, as well asdetermining 512, based upon comparing 510, whether the subjectcharacteristic substantially matches at least one sample characteristicto identify one or more subject sleep events occurring during the sleepinterval.

Process 500 may include additional, fewer, and/or alternative steps,including those described elsewhere herein. For instance, in someembodiments, process 500 may include generating, based upon theidentified one or more subject sleep events, a notification describingthe one or more sleep events, and transmitting the notification to theclient device. Process 500 may include determining a duration of eachidentified subject sleep event. Process 500 may include determining thata first subject sleep event recurs during the sleep interval, anddetermining a number of occurrences of the first subject sleep event. Insome embodiments, process 500 may include determining a number ofsubject sleep events that occurred during the sleep interval, anddetermining a level of sleep quality associated with the sleep intervalbased upon the number of subject sleep events. In some embodiments,process 500 may further include identifying one or more subject sleepevents occurring over a plurality of sleep intervals based uponrespective subject sound data for each sleep interval of the pluralityof sleep intervals, identifying at least one sleep trend associated withthe plurality of sleep intervals, generating a notification describingthe at least one sleep trend, and transmitting the notification to theclient device.

In some embodiments, receiving 504 subject sound data may includereceiving the subject sound data from at least one audio sensorassociated with the client device. In some embodiments, receiving 504subject sound data may include receiving the subject sound data from theclient device positioned at a location within a threshold distance ofthe user during the sleep interval.

Exemplary Embodiments & Functionality

In one aspect, a computer-based method for assessing sound to analyze auser's sleep is described. The method may be performed using a soundanalysis computing device including at least one processor incommunication with at least one memory device. The method may include:(i) storing sample sound data associated with a plurality of samplesleep events, the sample sound data including a plurality of samplecharacteristics each respectively associated with at least one samplesleep event of the plurality of sample sleep events; (ii) receiving,from a client device, subject sound data collected during a sleepinterval; (iii) analyzing, using a machine learning algorithm, thesubject sound data collected during the sleep interval; (iv)identifying, based upon the analyzing, a subject characteristicassociated with the subject sound data; (v) comparing the subjectcharacteristic with the plurality of sample characteristics; and/or (vi)determining, based upon the comparing, whether the subjectcharacteristic substantially matches at least one sample characteristicto identify one or more subject sleep events occurring during the sleepinterval.

In another aspect, a computer system for assessing sound to analyze auser's sleep is described. The computer system includes a processor, anda non-transitory, tangible, computer-readable storage medium havinginstructions stored thereon that, in response to execution by theprocessor, cause the processor to perform operations including: (i)storing sample sound data associated with a plurality of sample sleepevents, the sample sound data including a plurality of samplecharacteristics each respectively associated with at least one samplesleep event of the plurality of sample sleep events; (ii) receiving,from a client device, subject sound data collected during a sleepinterval; (iii) analyzing, using a machine learning algorithm, thesubject sound data collected during the sleep interval; (iv)identifying, based upon the analyzing, a subject characteristicassociated with the subject sound data; (v) comparing the subjectcharacteristic with the plurality of sample characteristics; and/or (vi)determining, based upon the comparing, whether the subjectcharacteristic substantially matches at least one sample characteristicto identify one or more subject sleep events occurring during the sleepinterval.

In yet another aspect, a non-transitory computer readable medium thatincludes executable instructions for assessing sound to analyze a user'ssleep is described. When executed by a sound analysis computing deviceincluding at least one processor in communication with at least onememory device, the computer executable instructions may cause the soundanalysis computing device to: (i) store sample sound data associatedwith a plurality of sample sleep events, the sample sound data includinga plurality of sample characteristics each respectively associated withat least one sample sleep event of the plurality of sample sleep events;(ii) receive, from a client device, subject sound data collected duringa sleep interval; (iii) analyze, using a machine learning algorithm, thesubject sound data collected during the sleep interval; (iv) identify,based upon the analysis, a subject characteristic associated with thesubject sound data; (v) compare the subject characteristic with theplurality of sample characteristics; and/or (vi) determine, based uponthe comparison, whether the subject characteristic substantially matchesat least one sample characteristic to identify one or more subject sleepevents occurring during the sleep interval.

Machine Learning & Other Matters

The computer-implemented methods discussed herein may includeadditional, less, or alternate actions, including those discussedelsewhere herein. The methods may be implemented via one or more localor remote processors, transceivers, servers, and/or sensors (such asprocessors, transceivers, servers, and/or sensors mounted on vehicles ormobile devices, or associated with smart infrastructure or remoteservers), and/or via computer-executable instructions stored onnon-transitory computer-readable media or medium.

Additionally, the computer systems discussed herein may includeadditional, less, or alternate functionality, including that discussedelsewhere herein. The computer systems discussed herein may include orbe implemented via computer-executable instructions stored onnon-transitory computer-readable media or medium.

A processor or a processing element may be trained using supervised orunsupervised machine learning, and the machine learning program mayemploy a neural network, which may be a convolutional neural network, adeep learning neural network, or a combined learning module or programthat learns in two or more fields or areas of interest. Machine learningmay involve identifying and recognizing patterns in existing data inorder to facilitate making predictions for subsequent data. Models maybe created based upon example inputs in order to make valid and reliablepredictions for novel inputs.

Additionally or alternatively, the machine learning programs may betrained by inputting sample data sets or certain data into the programs,such as sample audio files of known sounds. The machine learningprograms may utilize deep learning algorithms that may be primarilyfocused on audio pattern recognition, and may be trained afterprocessing multiple examples. The machine learning programs may includeBayesian program learning (BPL), audio/sound/voice recognition andsynthesis, Hidden Markov Models (HMM), vector quantization, featureextraction, k-means algorithms, natural language processing, semanticanalysis, automatic reasoning, kernel density estimation of probability(KDE), and/or other machine learning techniques—either individually orin combination.

In supervised machine learning, a processing element may be providedwith example inputs and their associated outputs, and may seek todiscover a general rule that maps inputs to outputs, so that whensubsequent novel inputs are provided the processing element may, basedupon the discovered rule, accurately predict the correct output. Inunsupervised machine learning, the processing element may be required tofind its own structure in unlabeled example inputs.

Additional Considerations

As will be appreciated based upon the foregoing specification, theabove-described embodiments of the disclosure may be implemented usingcomputer programming or engineering techniques including computersoftware, firmware, hardware or any combination or subset thereof. Anysuch resulting program, having computer-readable code means, may beembodied or provided within one or more computer-readable media, therebymaking a computer program product, i.e., an article of manufacture,according to the discussed embodiments of the disclosure. Thecomputer-readable media may be, for example, but is not limited to, afixed (hard) drive, diskette, optical disk, magnetic tape, semiconductormemory such as read-only memory (ROM), and/or any transmitting/receivingmedium, such as the Internet or other communication network or link. Thearticle of manufacture containing the computer code may be made and/orused by executing the code directly from one medium, by copying the codefrom one medium to another medium, or by transmitting the code over anetwork.

These computer programs (also known as programs, software, softwareapplications, “apps”, or code) include machine instructions for aprogrammable processor, and can be implemented in a high-levelprocedural and/or object-oriented programming language, and/or inassembly/machine language. As used herein, the terms “machine-readablemedium” “computer-readable medium” refers to any computer programproduct, apparatus and/or device (e.g., magnetic discs, optical disks,memory, Programmable Logic Devices (PLDs)) used to provide machineinstructions and/or data to a programmable processor, including amachine-readable medium that receives machine instructions as amachine-readable signal. The “machine-readable medium” and“computer-readable medium,” however, do not include transitory signals.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

As used herein, a processor may include any programmable systemincluding systems using micro-controllers, reduced instruction setcircuits (RISC), application specific integrated circuits (ASICs), logiccircuits, and any other circuit or processor capable of executing thefunctions described herein. The above examples are example only, and arethus not intended to limit in any way the definition and/or meaning ofthe term “processor.”

As used herein, the terms “software” and “firmware” are interchangeable,and include any computer program stored in memory for execution by aprocessor, including RAM memory, ROM memory, EPROM memory, EEPROMmemory, and non-volatile RAM (NVRAM) memory. The above memory types areexample only, and are thus not limiting as to the types of memory usablefor storage of a computer program.

In one embodiment, a computer program is provided, and the program isembodied on a computer readable medium. In an exemplary embodiment, thesystem is executed on a single computer system, without requiring aconnection to a sever computer. In a further embodiment, the system isbeing run in a Windows® environment (Windows is a registered trademarkof Microsoft Corporation, Redmond, Wash.). In yet another embodiment,the system is run on a mainframe environment and a UNIX® serverenvironment (UNIX is a registered trademark of X/Open Company Limitedlocated in Reading, Berkshire, United Kingdom). The application isflexible and designed to run in various different environments withoutcompromising any major functionality.

In some embodiments, the system includes multiple components distributedamong a plurality of computing devices. One or more components may be inthe form of computer-executable instructions embodied in acomputer-readable medium. The systems and processes are not limited tothe specific embodiments described herein. In addition, components ofeach system and each process can be practiced independent and separatefrom other components and processes described herein. Each component andprocess can also be used in combination with other assembly packages andprocesses. The present embodiments may enhance the functionality andfunctioning of computers and/or computer systems.

As used herein, an element or step recited in the singular and precededby the word “a” or “an” should be understood as not excluding pluralelements or steps, unless such exclusion is explicitly recited.Furthermore, references to “example embodiment” or “one embodiment” ofthe present disclosure are not intended to be interpreted as excludingthe existence of additional embodiments that also incorporate therecited features.

The patent claims at the end of this document are not intended to beconstrued under 35 U.S.C. § 112(f) unless traditionalmeans-plus-function language is expressly recited, such as “means for”or “step for” language being expressly recited in the claim(s).

This written description uses examples to disclose the disclosure,including the best mode, and also to enable any person skilled in theart to practice the disclosure, including making and using any devicesor systems and performing any incorporated methods. The patentable scopeof the disclosure is defined by the claims, and may include otherexamples that occur to those skilled in the art. Such other examples areintended to be within the scope of the claims if they have structuralelements that do not differ from the literal language of the claims, orif they include equivalent structural elements with insubstantialdifferences from the literal languages of the claims.

1-20. (canceled)
 21. A computer-based method for assessing sound toanalyze a user's sleep, the method performed using one or moreprocessors, the method comprising: receiving subject sound datacollected during a sleep interval; analyzing, using a machine learningalgorithm, the subject sound data collected during the sleep interval;identifying, based upon the analyzing, a subject characteristicassociated with the subject sound data; comparing the subjectcharacteristic with a plurality of sample characteristics, the pluralityof sample characteristics each being associated with at least one samplesleep event of a plurality of sample sleep events; and identifying,based upon the comparing, to identify one or more subject sleep eventsoccurring during the sleep interval.
 22. The method of claim 21, whereinthe subject characteristic includes at least one selected from a groupconsisting of a frequency characteristic, an amplitude characteristic, awaveform characteristic, and a duration characteristic.
 23. The methodof claim 21, further comprising: receiving driving behavior data of theuser; and correlating the driving behavior data with the one or moreidentified subject sleep events.
 24. The method of claim 21, furthercomprising: analyzing additional subject sound data collected during atime interval to determine that the user is falling asleep; determiningthe user is performing a task during the time interval; and generatingan alarm to prevent the user from falling asleep.
 25. The method ofclaim 21 further comprising: generating, based upon the one or moreidentified subject sleep events, a notification describing the one ormore identified subject sleep events; and transmitting the notificationto a client device.
 26. The method of claim 25 further comprising:determining a duration of one identified subject sleep event of the oneor more identified subject sleep events; wherein the notificationincludes an indication of a duration of the one identified subject sleepevent.
 27. The method of claim 25, wherein the notification includes anindication of at least one selected from a group consisting of a date ofthe sleep interval, a start time of the sleep interval, an end time ofthe sleep interval, and environmental information of the sleep interval.28. The method of claim 21 further comprising: determining a number ofsubject sleep events that occurred during the sleep interval; anddetermining a level of sleep quality associated with the sleep intervalbased upon the number of subject sleep events.
 29. The method of claim21 further comprising: identifying one or more subject sleep eventsoccurring over a plurality of sleep intervals based upon respectivesubject sound data for each sleep interval of the plurality of sleepintervals; identifying at least one sleep trend associated with theplurality of sleep intervals; generating a notification describing theat least one sleep trend; and transmitting the notification to a clientdevice.
 30. A computer system for assessing sound to analyze a user'ssleep, the computer system comprising: one or more processors; and oneor more memories having instructions stored thereon that, in response toexecution by the processor, cause the one or more processors to performoperations comprising: receiving subject sound data collected during asleep interval; analyzing, using a machine learning algorithm, thesubject sound data collected during the sleep interval; identifying,based upon the analyzing, a subject characteristic associated with thesubject sound data; comparing the subject characteristic with aplurality of sample characteristics, the plurality of samplecharacteristics each respectively associated with at least one samplesleep event of a plurality of sample sleep events; and identifying,based upon the comparing, one or more subject sleep events occurringduring the sleep interval.
 31. The computer system of claim 30, whereinthe subject characteristic includes at least one selected from a groupconsisting of a frequency characteristic, an amplitude characteristic, awaveform characteristic, and a duration characteristic.
 32. The computersystem of claim 30, wherein the operations further comprise: receivingdriving behavior data of the user; and correlating the driving behaviordata with the one or more identified subject sleep events.
 33. Thecomputer system of claim 30, wherein the operations further comprise:analyzing additional subject sound data collected during a time intervalto determine that the user is falling asleep; determining the user isperforming a task during the time interval; and generating an alarm toprevent the user from falling asleep.
 34. The computer system of claim30, wherein the operations further comprise: generating, based upon theone or more identified subject sleep events, a notification describingthe one or more identified subject sleep events; and transmitting thenotification to a client device.
 35. The computer system of claim 34,wherein the operations further comprise: determining a duration of oneidentified subject sleep event of the one or more identified subjectsleep events; wherein the notification includes an indication of theduration of the one identified subject sleep event.
 36. The computersystem of claim 30, wherein the operations further comprise: determiningthat a first subject sleep event recurs during the sleep interval; anddetermining a number of occurrences of the first subject sleep event.37. The computer system of claim 30, wherein the processor is furtherconfigured to perform operations comprising: identifying one or moresubject sleep events occurring over a plurality of sleep intervals basedupon respective subject sound data for each sleep interval of theplurality of sleep intervals; identifying at least one sleep trendassociated with the plurality of sleep intervals; generating anotification describing the at least one sleep trend; and transmittingthe notification to a client device.
 38. A non-transitory computerreadable medium that includes executable instructions for assessingsound to analyze a user's sleep, wherein when executed by one or moreprocessors, the executable instructions cause the one or more processorsto: receive subject sound data collected during a sleep interval;analyze, using a machine learning algorithm, the subject sound datacollected during the sleep interval; identify, based upon the analysis,a subject characteristic associated with the subject sound data; comparethe subject characteristic with a plurality of sample characteristics,the plurality of sample characteristics each respectively associatedwith at least one sample sleep event of a plurality of sample sleepevents; and identify, based upon the comparison, one or more subjectsleep events occurring during the sleep interval.
 39. The non-transitorycomputer readable medium of claim 38, wherein the executableinstructions further cause the one or more processors to: receivedriving behavior data of the user; and correlate the driving behaviordata with the one or more identified subject sleep events.
 40. Thenon-transitory computer readable medium of claim 38, wherein theexecutable instructions further cause the one or more processors to:analyze additional subject sound data collected during a time intervalto determine that the user is falling asleep; determine the user isperforming a task during the time interval; and generate an alarm toprevent the user from falling asleep